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Drawing Description Text - DRTX (8): 

FIG. 7 is a process flow diagram illustrating a process implemented on each 
node for determining whether to transmit locally generated packets downstream 
on the network and whether to transmit downstream available bandwidth 
information upstream . 

Detailed Description Text - DETX (23): 

The ring networks 52, 54, 56, etc. in one embodiment of the invention are 
bi-directional rings that transfer packets in both an upstream direction and a 
downstream direction. The ring networks connecting nodes 42 together can be 
implemented over Wide Area Network (WAN) links and/or Local Area Network (LAN) 
links and/or any other bus or backplane interconnection. 



Detailed Description Text - DETX (27): 

Two types of information are sent on the rings, data and control 
information. Preferably, as data is sent downstream in a clockwise direction 
by the outer ring, control information for managing the data traffic on the 
outer ring is sent upstream or counterclockwise on the inner ring. Likewise, 
as data packets are sent downstream in a counter clockwise direction on the 
inner ring, control information for management traffic on the inner ring are 
sent upstream or clockwise on the outer ring. Thus, each ring uses the other 
to send control information in a direction that is opposite to the direction 
that data packets are traveling. 



Detailed Description Text - DETX (28): 

In one embodiment, separate control packets are not generated for the 
control information. Instead, control information for the other ring is 
piggybacked in each data packet as part of a packet header. If no data packets 
are being sent on one ring, then a special empty packet is generated so that 
control information is sent upstream for the other ring. Separate control 
packets may also be generated. For purposes of this discussion, whenever a 
control packet is referred to, it should be understood that a special control 
packet or part of a data packet, such as a space reserved in a data packet 
header, could also be used. Control information is carried upstream to nodes 
about the bandwidth that is available to other nodes downstream . As is 
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described below, this information is used by each node to allocate to itself a 
fair amount of bandwidth for transmitting locally generated messages. In this 
context, a fair amount of bandwidth does not necessarily mean an exactly equal 
amount of bandwidth to other nodes, although it will be close to equal. The 
bandwidth is fairly determined by each node. 



Detailed Description Text - DETX (29): 

Each node implements the spatial reuse protocol by determining a fair amount 
of allocated bandwidth based on the bandwidth available to downstream nodes. 
Also, each node determines when control information should be sent upstream 
indicating that the node or nodes downstream are not receiving enough 
bandwidth. This is done by keeping track of four quantities: local transmit 
usage, downstream usage, allocated usage, and forward rate. 



Detailed Description Text - DETX (39): 

The description above shows how each node determines a fair amount of 
allocated usage for the node. When downstream nodes have limited bandwidth 
available for transmitting data, a message is sent upstream indicating the 
amount of bandwidth available to the downstream nodes. Each node evaluates 
control information received from downstream nodes and decreases its allocated 
usage if it has allocated more bandwidth to itself than downstream nodes are 
currently receiving. Before describing how each node uses the allocated 
bandwidth to determine whether or not to send data, it will be useful to 
consider an example of how network bandwidth is shared in a fair and efficient 
manner with spatial and local reuse when the nodes implement the spatial reuse 
protocol described above. 



Detailed Description Text - DETX (40): 

Returning to FIG. 5, consider a case where node 202 is transmitting to node 
204 using links 212 and 213 on the outer ring. If no information is received 
by node 202 indicating downstream available bandwidth, the allocated usage for 
node 202 increases until it reaches the maximum allowed allocated usage . Thus, 
links 212 and 213 become completely devoted to carrying data transmitted by 
node 202. If node 203 needs to transmit to node 204, it immediately notices 
that it is forwarding data at a rate that prevents it from transmitting any of 
its own data onto the network. It has a very low local transmit usage and 
since its local transmit usage is less than its forward rate, it sends 
information upstream to node 202 indicating the available bandwidth for node 
203. When node 202 receives indication of a low downstream available 
bandwidth, it decreases its allocated usage and reduces amount of its own data 
that it is transmitting on the network. Then, node 203 is able to transmit 
some of its own data onto the network. 



Detailed Description Text - DETX (42): 

The forward rate is used by each node to determine whether or not downstream 
available bandwidth information should be forwarded upstream . Since, as 
described above, downstream allocated bandwidth is used by nodes to reduce the 
amount of bandwidth that is allocated for transmitting locally generated data, 
it is desirable that downstream allocated bandwidth not be forwarded to nodes 
that are not contributing to congestion. This is accomplished by having nodes 
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only forward information indicating downstream available bandwidth if the local 
transmit usage of the node is less than the forward rate for the node. Thus, 
when a node receives information indicating downstream allocated bandwidth, the 
node determines whether or not its local transmit usage exceeds its forward 
rate. If its local transmit usage exceeds the forward rate, then the traffic 
congestion experienced downstream is caused primarily by that node and not by 
upstream nodes. Therefore, the node does not forward the downstream allocated 
bandwidth upstream . This enables the network to efficiently achieve local 
reuse. 



Detailed Description Text - DETX (46): 

It should be noted that decisions whether or not to send information 
upstream indicating downstream available bandwidth are made by each node when 
the node has locally generated data that it needs to transmit on the network. 
The following description will show how each individual node determines whether 
it can send data on the network based on the state of its transit buffer and 
its allocated usage. 



Detailed Description Text - DETX (48): 

Actually, the node checks whether the amount of data stored in the transit 
buffer is less than a certain threshold. The threshold is usually some amount 
below the full capacity of the transit buffer to allow for a safety factor to 
prevent packets from being dropped that are supposed to be forwarded. 
Additionally, in one embodiment, the threshold is further reduced to allow for 
priority data. When the node wants to transmit its own packets onto the 
network from the packet transmission buffer and it determines that its ability 
to do so is limited by packets being forwarded from upstream nodes through the 
transit buffer, then that node piggybacks information indicating the amount of 
bandwidth that is available to the node on a packet that is sent upstream. In 
one embodiment, space is reserved in packet headers for downstream available 
bandwidth information. By indicating its local transmit usage to upstream 
nodes that implement the spatial reuse protocol, the downstream node insures 
that it will eventually have available to it a fair amount of network 
bandwidth. The process by which a node determines whether or not to send 
downstream available bandwidth information upstream and whether or not to 
transmit its own packets when the transit buffer is below the threshold is 
further described below. 



Detailed Description Text - DETX (49): 

FIG. 7 is a process flow diagram illustrating a process implemented on each 
node for determining whether to transmit locally generated packets downstream 
on the network and whether to transmit downstream available bandwidth 
information upstream . The process starts at 400. In a step 402 the node 
determines whether or not the transit buffer has a high priority packet in it. 
If it does, then control is transferred to a step 404 where the node determines 
whether or not its forward rate is greater than its allocated usage, if the 
forward rate is greater than its allocated usage, then control is transferred 
to a step 406 and a flag is set indicating that a notification of downstream 
available bandwidth should be forwarded upstream . As described above, when 
upstream packets are sent, downstream available bandwidth is then included with 
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the packets in the packet header. The time that the flag remains set may be 
adjusted to tune performance. 

Detailed Description Text - DETX (57): 

Thus, the node regulates the amount of it own traffic that it transmits on 
the network according to its allocated usage and the node sends information 
upstream indicating downstream available bandwidth when local transmit usage is 
less than allocated usage. 



Claims Text - CLTX (1): 

1 . A multidimensional ring network, comprising: a plurality of nodes 
arranged into a plurality of node sets; multiple ring networks each coupling 
three or more of the nodes in one of the node sets together; the multiple ring 
networks connected together to form a logical 3 or more dimensional ring matrix 
network where each dimension of the 3 or more dimensional ring matrix network 
includes three or more of the ring networks and the nodes at each logical edge 
of the ring matrix network are coupled directly to the nodes at each opposite 
corresponding edge of the ring matrix network; network interfaces for each one 
of the nodes configured to send and receive packets in an upstream direction 
and a downstream direction on the ring networks; and packet controllers on 
each one of the nodes configured to allocate a bandwidth for locally generated 
network packets sent in the downstream direction wherein the packet controller 
determines a minimum downstream available network bandwidth available in the 
downstream direction from information received in the upstream direction and 
adjusts the local allocated bandwidth based on the minimum downstream available 
network bandwidth, periodically increasing the local allocated bandwidth toward 
a maximum local allocated bandwidth and wherein the packet controller uses the 
local allocated bandwidth to govern whether a class of locally generated 
network packets are sent in the downstream direction. 



Claims Text - CLTX (2): 

2. A ring architecture, comprising: multiple network processing nodes 
logically arranged along multiple axes; multiple ring networks each connecting 
together at least three of the network processing nodes that are logically 
arranged along a same axes, each one of the multiple ring networks connecting 
all of the nodes on the same axes together in sequence and connecting the nodes 
on opposite ends of the same axes directly together in a ring configuration, at 
least three of the ring networks logically coupling the network processing 
nodes together along at least three first parallel axes and at least three 
additional ring networks logically coupling the same network processing nodes 
together logically along at least three second axes perpendicular to the first 
axes to form a first at least 3.times.3 lattice interconnected ring network 
where each one of the network processing nodes on the ends of the lattice are 
directly coupled to the network processing nodes on corresponding opposite ends 
of the lattice; additional at least 3.times.3 lattice interconnected ring 
networks formed along each axis of the first lattice interconnected ring 
network; network interfaces for each one of the nodes configured to send and 
receive packets in an upstream direction and a downstream direction on the ring 
networks; and packet controllers on each one of the nodes configured to 
allocate a bandwidth for locally generated network packets sent in the 
downstream direction wherein the packet controller determines a minimum 
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downstream available network bandwidth available in the downstream direction 
from information received in the upstream direction and adjusts the local 
allocated bandwidth based on the minimum downstream available network 
bandwidth, periodically increasing the local allocated bandwidth toward a 
maximum local allocated bandwidth and wherein the packet controller uses the 
local allocated bandwidth to govern whether a class of locally generated 
network packets are sent in the downstream direction. 
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Detailed Description Text - DETX (25): 

In step 330, an amount of usage is allowed based on the assigned usage rate 
and the reserve. If the excess of the desired usage rate over the assigned 
usage rate corresponds to a usage less than or equal to the reserve, then the 
amount of usage allowed is the amount eguivalent to the desired usage rate. 
Otherwise, the amount of usage allowed is limited by the assigned usage rate 
and the reserve. Control then passes to step 340. 



Detailed Description Text - DETX (26): 

In step 340, the reserve is decremented. The reserve is decremented by the 
amount of the reserve used to supplement the allowed usage in step 330. Note 
that the reserve is never decremented below zero because the amount of the 
reserve used to supplement the amount of usage allowed never exceeds the amount 
' of data in the reserve. Execution of the steps ends until the steps are 
invoked again. 



Detailed Description Text - DETX (43): 

Table 430 shows various states after each access by non-real-time client 400 
of shared resource 428. The columns from left to right are the time interval, 
the desired usage rate, the assigned usage rate, the allowed usage, the 
increment or decrement of the reserve, and the reserve after execution of the 
steps in each time interval. The columns representing amounts of data contain 
numbers expressing a rate in blocks/sec. In the increment or decrement column, 
increments are represented by positive integers, and decrements are represented 
by negative integers. 



Detailed Description Text - DETX (45): 

In step 350, the usage rate allowed is the desired usage rate, which is 10 
blocks/sec. Therefore, non-real-time client 400 accesses shared resource 428 at 
a rate of 10 blocks/sec. At step 360, a determination is made of whether the 
desired usage rate is less than the assigned usage rate. Because the desired 
usage rate is equal to the assigned usage rate of data, the reserve is not 
incremented and execution of the steps ceases. 
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Detailed Description Text - DETX (54): 

In step 330, an amount of usage is allowed based on the assigned usage rate 
and the reserve. The difference between the desired usage rate and the 
assigned usage rate is 2 blocks/sec. The reserve contains 5 blocks of data. 
Based on the period of time between the time intervals, which is 1 second, the 
difference is the equivalent of 2 blocks. Because the difference of 2 blocks 
is less than the reserve of 5 blocks, the allowed usage rate is the desired 
rate of 12 blocks/sec. Control then passes to step 340. 



Detailed Description Text - DETX (57): 

In step 330, an amount of usage is allowed based on the assigned usage rate 
and the reserve. The difference between the desired usage rate and the 
assigned usage rate is 4 blocks/sec. Based on the period of time between the 
time intervals, which is 1 second, the difference is the equivalent of 4 
blocks. The reserve contains 3 blocks of data. Because the difference of 4 
blocks is greater than the reserve at 3 blocks, only the amount in the reserve 
is used to provide the amount of usage allowed. The usage allowed is 13 
blocks, which is less than that which would be provided at the desired usage 
rate of 14 blocks/sec. Control then passes to step 340. 
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